{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to the Research Environment\n",
    "\n",
    "The research environment is powered by IPython notebooks, which allow one to perform a great deal of data analysis and statistical validation. We'll demonstrate a few simple techniques here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 研究环境的介绍\n",
    "这是一个基于jupyter notebook(原IPython notebook)的研究环境，能出色的完成大数据的分析和统计验证。我们将在这里做一些简单的技巧演示。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Code Cells vs. Text Cells\n",
    "\n",
    "As you can see, each cell can be either code or text. To select between them, choose from the 'Cell Type' dropdown menu on the top left."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 代码单元格 vs 文本单元格\n",
    "如你所见，单元格可以是代码编辑模式，也可以是文本编辑模式。可用通过点击顶部菜单的`Cell`的下拉菜单中`Cell Type`来进行选择。\n",
    "亦可通过顶部菜单中下拉菜单进行选择：`Code`即为代码编辑模式，`Markdown`即为文本编辑模式。如下图所示:\n",
    "\n",
    "![image](image/cell_mode_change.jpg)\n",
    "\n",
    "在单元格未进入编辑时，可以通过快捷键进行切换，`Y`切换成代码模式，`M`切换成文本编辑模式，当单元格处于编辑状态时，可以按`Esc`退出编辑模式， 按`Enter`进入编辑模式。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Executing a Command\n",
    "\n",
    "A code cell will be evaluated when you press play, or when you press the shortcut, shift-enter. Evaluating a cell evaluates each line of code in sequence, and prints the results of the last line below the cell."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 执行一个命令\n",
    "当处于`代码编辑`模式时，通过快捷键`Shift`+`Enter`来逐行执行单元格中代码，并会打印出单元格最后一行代码执行的结果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "2 + 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "有时候并没有打印结果，比如说赋值的时候."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X = 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "记住，只有最后一行的代码执行的结果会被打印出来."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "2 + 2\n",
    "3 + 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "你可以通过`print`来打印你想要的代码结果."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "print(2 + 2)\n",
    "3 + 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Knowing When a Cell is Running\n",
    "\n",
    "While a cell is running, a `[*]` will dsiplay on the left. When a cell has yet to be executed, `[ ]` will display. When it has been run, a number will display indicating the order in which it was run during the execution of the notebook `[5]`. Try on this cell and note it happening."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 单元格的状态\n",
    "当一个单元格左侧的标记为`[*]`时，表明程序正在运行.`[ ]`表明单元格还未执行.当一个单元格运行完毕后，会在这对方括号中加上一个数字，表示已完成的状态，比如`[5]`.\n",
    "- 特别的说明:当你重新打开notebook,或者重启kernel(并未清理结果)之后，单元格左侧的方框虽然有数字，但其实并未运行."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 花点时间去运行一段代码\n",
    "c = 0\n",
    "for i in range(10000000):\n",
    "    c = c + i\n",
    "c"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing Libraries\n",
    "\n",
    "The vast majority of the time, you'll want to use functions from pre-built libraries. You can't import every library on Quantopian due to security issues, but you can import most of the common scientific ones. Here I import numpy and pandas, the two most common and useful libraries in quant finance. I recommend copying this import statement to every new notebook.\n",
    "\n",
    "Notice that you can rename libraries to whatever you want after importing. The `as` statement allows this. Here we use `np` and `pd` as aliases for `numpy` and `pandas`. This is a very common aliasing and will be found in most code snippets around the web. The point behind this is to allow you to type fewer characters when you are frequently accessing these libraries."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 导入库\n",
    "绝大多数时候，我们需要使用预构建库(pre-built libraries)中的函数。出于安全考虑(基于平台的角度)，我们无法导入Quantopian上的每个库，但是可以导入最常见，用于科学计算的库。这里我将会导入`numpy`和`pandas`，两个在量化金融中常见且实用的库。\n",
    "我建议将这个导入代码复制到每一个新的notebook.\n",
    "\n",
    "你可以使用`as`关键字将导入的库重命名，我们将使用 `np`和`pd`分别作为`numpy`和`pandas`的别名，这算是全世界大家公认的别名。使用别名的好处就是当你频繁使用者这些库时，能输入较少的字符。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from scipy.io import loadmat\n",
    "# 这是一个在画图方面非常优秀的库\n",
    "import matplotlib.pyplot as plt\n",
    "import datetime\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tab Autocomplete\n",
    "\n",
    "Pressing tab will give you a list of IPython's best guesses for what you might want to type next. This is incredibly valuable and will save you a lot of time. If there is only one possible option for what you could type next, IPython will fill that in for you. Try pressing tab very frequently, it will seldom fill in anything you don't want, as if there is ambiguity a list will be shown. This is a great way to see what functions are available in a library.\n",
    "\n",
    "Try placing your cursor after the `.` and pressing tab."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tab键 代码自动补全\n",
    "当你按下`Tab`键时，会自动出现一个猜测你下一步，你可能输入内容的列表。这个功能很实用而且能省不少事。如果只有一个可能选项，IPython则会自动帮助你完成输入。请频繁的按下`Tab`吧，它几乎不会出现你不想要的内容，一如这里有歧义的话，就会出现一个列表。想看某个库中有哪些函数，这是一个不错的方式。\n",
    "\n",
    "请将鼠标的置于`np.random.`中最后一个`.`之后，并按下`Tab`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.random."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting Documentation Help\n",
    "\n",
    "Placing a question mark after a function and executing that line of code will give you the documentation IPython has for that function. It's often best to do this in a new cell, as you avoid re-executing other code and running into bugs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 获取文档帮助\n",
    "在IPython中，在一个函数后面加上`?`并执行该代码，就会获得该函数的文本帮助。建议通常在一个新的单元格执行，这样能避免重新执行其他代码或者发生bug."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.random.normal?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sampling\n",
    "\n",
    "We'll sample some random data using a function from `numpy`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 生成样本\n",
    "我们将用`numpy`中的函数随机生成一些数据"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 译者注：通常计算机中的随机函数，都是伪随机，在`numpy`中，我们通过使用`np.random.seed()`函数来控制随机结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Sample 100 points with a mean of 0 and an std of 1. This is a standard normal distribution.\n",
    "# 随机生成100样本点，这些点服从均值为0，标准差为1的正态分布。\n",
    "np.random.seed(1) # 确保大家生成的样本保持一致\n",
    "\n",
    "X = np.random.normal(0, 1, 100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plotting\n",
    "\n",
    "We can use the plotting library we imported as follows."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 画图\n",
    "我们使用刚才导入的库进行画图\n",
    "\n",
    "`import matplotlib.pyplot as plt`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plt.plot(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Squelching Line Output\n",
    "\n",
    "You might have noticed the annoying line of the form `[<matplotlib.lines.Line2D at 0x7f72fdbc1710>]` before the plots. This is because the `.plot` function actually produces output. Sometimes we wish not to display output, we can accomplish this with the semi-colon as follows."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 让输出更纯净\n",
    "你可能注意到我们画的图前面有一行非常讨厌的输出：`[<matplotlib.lines.Line2D at 0x1c******>]`\n",
    "\n",
    "这是`.plot`函数输出结果，有时候我们并不希望他出现，我们可以加上`;`实现这个功能；"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plt.plot(X);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding Axis Labels\n",
    "\n",
    "No self-respecting quant leaves a graph without labeled axes. Here are some commands to help with that."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 添加坐标轴标签\n",
    "有追求的宽客是不会让图表没有坐标轴标签的。\n",
    "\n",
    "这里有一些命令可以帮我们实现他。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.random.seed(2)\n",
    "\n",
    "X = np.random.normal(0, 1, 100)\n",
    "X2 = np.random.normal(0, 1, 100)\n",
    "\n",
    "plt.plot(X);\n",
    "plt.plot(X2);\n",
    "plt.xlabel('Time') # 我们生成的数据是没有单位的，但不要忘记他的单位。\n",
    "plt.ylabel('Returns')\n",
    "plt.legend(['X', 'X2']);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating Statistics\n",
    "\n",
    "Let's use `numpy` to take some simple statistics."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 计算统计值\n",
    "让我们用`numpy`来做一些简单的统计计算。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.mean(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.std(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting Real Pricing Data\n",
    "\n",
    "Randomly sampled data can be great for testing ideas, but let's get some real data. We can use `get_pricing` to do that. You can use the `?` syntax as discussed above to get more information on `get_pricing`'s arguments."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 获取真实的价格数据\n",
    "\n",
    "随机生成的数据能很好检验我们的想法，但是我们最终也是需要真实的数据来进行检测。我们将用`get_pricing`函数提取数据，可以使用刚才我们提到`?`命令去获取更多关于`get_pricing`函数的参数信息。\n",
    "\n",
    "- 译者注：请注意`get_pricing`为quantopain函数，本地使用相关函数是无效的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# data = get_pricing('MSFT', start_date='2012-1-1', end_date='2015-6-1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 译者为大家提供一个本地的数据，以便完成后面的练习"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "file_name = r'data\\IF888-2011.mat'\n",
    "origin_data = loadmat(file_name)\n",
    "data = pd.DataFrame(origin_data['IF888'])\n",
    "data.columns = ['date', 'price']\n",
    "origin = np.datetime64('0000-01-01', 'D') - np.timedelta64(1, 'D')\n",
    "data['date'] = data['date'].map(lambda x: x  * np.timedelta64(1, 'D') + origin)\n",
    "data.index = data['date'].tolist()\n",
    "data = data.iloc[:,1:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our data is now a dataframe. You can see the datetime index and the colums with different pricing data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`data`是的数据格类型是`pandas`中的`dataframe`。可以通过`index`和`colums`来查看不同维度下价格数据"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a pandas dataframe, so we can index in to just get price like this. For more info on pandas, please [click here](http://pandas.pydata.org/pandas-docs/stable/10min.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这是一个pandas dataframe, 我们可以使用index去获取价格信息. 更多关于pandas的信息请[点击这里](http://pandas.pydata.org/pandas-docs/stable/10min.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X = data['price']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because there is now also date information in our data, we provide two series to `.plot`. `X.index` gives us the datetime index, and `X.values` gives us the pricing values. These are used as the X and Y coordinates to make a graph."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "因为这里有日期信息在我们的数据中，我们将`X.index`得到的日期索引数据，和`X.values`得到的价格数据作为`.plot`X和Y轴数据用于画图。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plt.plot(X.index, X.values)\n",
    "plt.ylabel('Price')\n",
    "plt.legend(['IF888'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们在真实数据上来做点统计计算。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.mean(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.std(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Getting Returns from Prices\n",
    "\n",
    "We can use the `pct_change` function to get returns. Notice how we drop the first element after doing this, as it will be `NaN` (nothing -> something results in a NaN percent change)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 根据价格数据计算回报\n",
    "我们用`pct_change`函数提取回报，注意：返回的第一个元素是`NaN`，所以我们会忽略掉他(因为第一个元素之前没有元素，无法比较，故返回一个NaN)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "R = X.pct_change()[1:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can plot the returns distribution as a histogram."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以用一个直方图去描述回报分布"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plt.hist(R, bins=20)\n",
    "plt.xlabel('Returns')\n",
    "plt.ylabel('Frequency')\n",
    "plt.grid(True)\n",
    "plt.legend(['IF888 Returns']);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get statistics again."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "再次计算统计量"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.mean(R) # the same as R.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "R.std() # the same sa np.std(R)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's go backwards and generate data out of a normal distribution using the statistics we estimated from Microsoft's returns. We'll see that we have good reason to suspect Microsoft's returns may not be normal, as the resulting normal distribution looks far different."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在我们用回报数据的均值和标准差作为一个正态分布的参数去生成数据，我们有理由相信`IF888`的回报并不服从正态分布，因为对比后发现，两者相去甚远。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.random.seed(3)\n",
    "\n",
    "plt.hist(np.random.normal(np.mean(R), np.std(R), 10000), bins=20)\n",
    "plt.xlabel('Returns')\n",
    "plt.ylabel('Frequency')\n",
    "plt.grid(True)\n",
    "plt.legend(['Normal Distribution Returns'], loc='best');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating a Moving Average\n",
    "\n",
    "`pandas` has some nice tools to allow us to generate rolling statistics. Here's an example. Notice how there's no moving average for the first 60 days, as we don't have 60 days of data on which to generate the statistic."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 生成一条移动均线\n",
    "`pandas`有非常优秀的工具能让我们生成rolling statistics.这里有个例子.注意我们是没有足够的数据去生成前60日均线数据的。 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Take the average of the last 60 days at each timepoint.\n",
    "MAVG = X.rolling(window=60).mean()\n",
    "plt.plot(X.index, X.values)\n",
    "plt.plot(MAVG.index, MAVG.values)\n",
    "plt.ylabel('Price')\n",
    "plt.legend(['IF888', '60-day MAVG']);"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 tensorflow",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
